Gesture-controlled augmented reality experience using a mobile communications device

ABSTRACT

A method for producing a gesture-controlled augmented reality experience using a first mobile communications device includes receiving a motion sensor input on a motion sensor input modality of the first mobile communications device, calculating a trajectory of a camera of the first mobile communications device in response to the received motion sensor input, receiving a visual input captured by the camera of the first mobile communications device, translating a gesture of a user into a set of quantified values based on the received visual input and the calculated trajectory of the camera, and controlling an augmented reality object within a three-dimensional virtual environment in response to a substantial match between the set of quantified values and a set of predefined values.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application relates to and claims the benefit of U.S.Provisional Application No. 62/443,461 filed Jan. 6, 2017 and entitled“AUGMENTED REALITY SCANNED AND CONTROLLED OBJECTS,” the entiredisclosure of which is hereby wholly incorporated by reference.

STATEMENT RE: FEDERALLY SPONSORED RESEARCH/DEVELOPMENT

Not Applicable

BACKGROUND 1. Technical Field

The present disclosure relates generally to human-computer interfacesand mobile devices, and more particularly, to gesture-controlledinteractions with a three-dimensional virtual environment.

2. Related Art

Mobile devices fulfill a variety of roles, from voice communications andtext-based communications such as Short Message Service (SMS) ande-mail, to calendaring, task lists, and contact management, as well astypical Internet based functions such as web browsing, socialnetworking, online shopping, and online banking. With the integration ofadditional hardware components, mobile devices can also be used forphotography or taking snapshots, navigation with mapping and GlobalPositioning System (GPS), cashless payments with NFC (Near FieldCommunications) point-of-sale terminals, and so forth. Such devices haveseen widespread adoption in part due to the convenient accessibility ofthese functions and more from a single portable device that can alwaysbe within the user's reach.

Although mobile devices can take on different form factors with varyingdimensions, there are several commonalities between devices that sharethis designation. These include a general purpose data processor thatexecutes pre-programmed instructions, along with wireless communicationmodules by which data is transmitted and received. The processor furthercooperates with multiple input/output devices, including combinationtouch input display screens, audio components such as speakers,microphones, and related integrated circuits, GPS modules, and physicalbuttons/input modalities. More recent devices also includeaccelerometers and compasses that can sense motion and direction. Forportability purposes, all of these components are powered by an on-boardbattery. In order to accommodate the low power consumption requirements,ARM architecture processors have been favored for mobile devices.Several distance and speed-dependent communication protocols may beimplemented, including longer range cellular network modalities such asGSM (Global System for Mobile communications), CDMA, and so forth, highspeed local area networking modalities such as WiFi, and close rangedevice-to-device data communication modalities such as Bluetooth.

Management of these hardware components is performed by a mobileoperating system, also referenced in the art as a mobile platform. Themobile operating system provides several fundamental software modulesand a common input/output interface that can be used by third partyapplications via application programming interfaces.

User interaction with the mobile device, including the invoking of thefunctionality of these applications and the presentation of the resultstherefrom, is, for the most part, restricted to the graphical touch userinterface. That is, the extent of any user interaction is limited towhat can be displayed on the screen, and the inputs that can be providedto the touch interface are similarly limited to what can be detected bythe touch input panel. Touch interfaces in which users tap, slide,flick, pinch regions of the sensor panel overlaying the displayedgraphical elements with one or more fingers, particularly when coupledwith corresponding animated display reactions responsive to suchactions, may be more intuitive than conventional keyboard and mouseinput modalities associated with personal computer systems. Thus,minimal training and instruction is required for the user to operatethese devices.

However, mobile devices must have a small footprint for portabilityreasons. Depending on the manufacturer's specific configuration, thescreen may be three to five inches diagonally. One of the inherentusability limitations associated with mobile devices is the reducedscreen size; despite improvements in resolution allowing for smallerobjects to be rendered clearly, buttons and other functional elements ofthe interface nevertheless occupy a large area of the screen.Accordingly, notwithstanding the enhanced interactivity possible withmulti-touch input gestures, the small display area remains a significantrestriction of the mobile device user interface. This limitation isparticularly acute in graphic arts applications, where the canvas iseffectively restricted to the size of the screen. Although the logicalcanvas can be extended as much as needed, zooming in and out whileattempting to input graphics is cumbersome, even with the larger tabletform factors.

Expanding beyond the confines of the touch interface, some appdevelopers have utilized the integrated accelerometer as an inputmodality. Some applications such as games are suited for motion-basedcontrols, and typically utilize roll, pitch, and yaw rotations appliedto the mobile device as inputs that control an on-screen element. In thearea of advertising, motion controls have been used as well. See, forexample, U.S. Patent Application Pub. No. 2015/0186944, the entirecontents of which is incorporated herein by reference. More recentremote controllers for video game console systems also have incorporatedaccelerometers such that motion imparted to the controller is translatedto a corresponding virtual action displayed on-screen.

Other video game console systems, such as Microsoft's Kinect, haveincorporated motion sensing input devices based on gesture recognition,with no physical contact between the player and the device.

Because motion is one of the most native forms of interaction betweenhuman beings and tangible objects, it would be desirable to utilize suchinputs to the mobile device for interactions between a user and athree-dimensional virtual environment.

BRIEF SUMMARY

The present disclosure contemplates various methods and devices forproducing a gesture-controlled augmented reality experience. Inaccordance with one embodiment, there is a method for producing agesture-controlled augmented reality experience using a first mobilecommunications device. The method includes receiving a motion sensorinput on a motion sensor input modality of the first mobilecommunications device, calculating a trajectory of a camera of the firstmobile communications device in response to the received motion sensorinput, receiving a visual input captured by the camera of the firstmobile communications device, translating a gesture of a user into a setof quantified values based on the received visual input and thecalculated trajectory of the camera, and controlling an augmentedreality object within a three-dimensional virtual environment inresponse to a substantial match between the set of quantified values anda set of predefined values.

The method may include displaying the augmented reality object on thefirst mobile communications device, which may include displaying amovable-window view of the three-dimensional virtual environment on thefirst mobile communications device.

The method may include outputting, on the first mobile communicationsdevice, at least one of visual, auditory, and haptic feedback inresponse to a substantial match between the set of quantified values anda set of predefined values.

The method may include receiving an auditory input captured by amicrophone of the first mobile communications device, and the step ofcontrolling may include controlling the augmented reality object withinthe three-dimensional virtual environment in response to the receivedauditory input. The method may further include detecting a signature ofmusic in response to the received auditory input, and the step ofcontrolling may include controlling the augmented reality object withinthe three-dimensional virtual environment in response to the detectedsignature.

The method may include transmitting object control data to a secondmobile communications device held by the user and communicativelycoupled to the first mobile communications device, the object controldata defining one or more movements of the augmented reality objectwithin the three-dimensional virtual environment in response to saidcontrolling. The method may include receiving a set of secondaryquantified values from the second mobile communications device, and thestep of controlling may include controlling the augmented reality objectwithin the three-dimensional virtual environment in response to asubstantial match between the set of secondary quantified values and theset of predefined values. The set of secondary quantified values may bederived from a motion sensor input on a motion sensor input modality ofthe second mobile communications device, a visual input captured by acamera of the second mobile communications device, and/or an auditoryinput captured by a microphone of the second mobile communicationsdevice.

The method may include transmitting feedback data to a second mobilecommunications device held by the user and communicatively coupled tothe first mobile communications device, the feedback data defining atleast one of visual, auditory, and haptic feedback to be output by thesecond mobile communications device in response to a substantial matchbetween the set of quantified values and a set of predefined values.

The motion sensor input modality of the first mobile communicationsdevice may include at least one of an accelerometer, a gyroscope, and amagnetometer integrated into the first mobile communications device.

In accordance with another embodiment, there is a system including anon-transitory program storage medium readable by a first mobilecommunications device, the medium tangibly embodying one or moreprograms of instructions executable by the device to perform operationsfor producing a gesture-controlled augmented reality experience. Theoperations include receiving a motion sensor input on a motion sensorinput modality of the first mobile communications device, calculating atrajectory of a camera of the first mobile communications device inresponse to the received motion sensor input, receiving a visual inputcaptured by the camera of the first mobile communications device,translating a gesture of a user into a set of quantified values based onthe received visual input and the calculated trajectory of the camera,and controlling an augmented reality object within a three-dimensionalvirtual environment in response to a substantial match between the setof quantified values and a set of predefined values.

The system may include the first mobile communications device, and thefirst mobile communications device may include a processor orprogrammable circuitry for executing the one or more programs ofinstructions. The system may further include a second mobilecommunications device held by the user and communicatively coupled tothe first mobile communications device, and the operations may furtherinclude transmitting object control data to the second mobilecommunications device, the object control data defining one or moremovements of the augmented reality object within the three-dimensionalvirtual environment in response to said controlling. The second mobilecommunications device may transmit a set of secondary quantified valuesto the first mobile communications device, and the controlling operationmay include controlling the augmented reality object within thethree-dimensional virtual environment in response to a substantial matchbetween the set of secondary quantified values and the set of predefinedvalues. The second mobile communications device may derive the set ofsecondary quantified values from a motion sensor input on a motionsensor input modality of the second mobile communications device, avisual input captured by a camera of the second mobile communicationsdevice, and/or an auditory input captured by a microphone of the secondmobile communications device.

In accordance with another embodiment, there is a mobile communicationsdevice operable to produce a gesture-controlled augmented realityexperience. The mobile communications device includes a motion sensorfor receiving a motion sensor input, a camera for capturing a visualinput, and a processor for calculating a trajectory of the camera inresponse to the received motion sensor input, translating a gesture of auser into a set of quantified values based on the received visual inputand the calculated trajectory of the camera, and controlling anaugmented reality object within a three-dimensional virtual environmentin response to a substantial match between the set of quantified valuesand a set of predefined values.

The mobile communications device may include a display, and theprocessor may control the display to display the augmented realityobject within a movable-window view of the three-dimensional virtualenvironment.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features and advantages of the various embodimentsdisclosed herein will be better understood with respect to the followingdescription and drawings, in which like numbers refer to like partsthroughout, and in which:

FIG. 1 illustrates one exemplary mobile communications device 10 onwhich various embodiments of the present disclosure may be implemented;

FIG. 2 illustrates one embodiment of a method for producing agesture-controlled augmented reality experience using the mobilecommunications device 10;

FIG. 3 shows an example of a gesture-controlled augmented realityexperience produced according to the method of FIG. 2;

FIGS. 4A and 4B relate to another example of a gesture-controlledaugmented reality experience produced according to the method of FIG. 2,of which FIG. 4A shows the display of an augmented reality object on themobile communications device 10 while the hands of the person in thescene are close together and FIG. 4B shows the display while the handsof the person in the scene are far apart;

FIGS. 5A-5F show additional aspects related to the example of FIGS. 4Aand 4B, of which FIG. 5A shows the user moving one hand around the sceneto trace a path, FIG. 5B shows the user holding two hands out withcupped palms facing upward, FIG. 5C shows the user holding two hands outwith cupped palms facing upward and more inward than in FIG. 5B, FIG. 5Dshows the user holding a single hand out palm up, FIG. 5E shows the userholding both hands out palms up, and FIG. 5F shows the user making agrabbing motion;

FIG. 6 illustrates one embodiment of a method for producing agesture-controlled augmented reality experience using a mobilecommunications device 10A in communication with an additional mobilecommunications device 10B;

FIG. 7 illustrates a complementary method to the method of FIG. 6, to beperformed by the mobile communications device 10B; and

FIGS. 8A-8C relate to an example of a gesture-controlled augmentedreality experience produced according to the method of FIG. 6 and thecomplementary method of FIG. 7, of which FIG. 8A shows the display of anaugmented reality object on the mobile communications device 10A, FIG.8B shows the person in the scene viewing a display of the augmentedreality object on the additional mobile communications device 10B, andFIG. 8C shows the person in the scene shaking the additional mobilecommunications device 10B.

DETAILED DESCRIPTION

The present disclosure encompasses various embodiments of methods anddevices for producing a gesture-controlled augmented reality experience.The detailed description set forth below in connection with the appendeddrawings is intended as a description of the several presentlycontemplated embodiments of these methods, and is not intended torepresent the only form in which the disclosed invention may bedeveloped or utilized. The description sets forth the functions andfeatures in connection with the illustrated embodiments. It is to beunderstood, however, that the same or equivalent functions may beaccomplished by different embodiments that are also intended to beencompassed within the scope of the present disclosure. It is furtherunderstood that the use of relational terms such as first and second andthe like are used solely to distinguish one from another entity withoutnecessarily requiring or implying any actual such relationship or orderbetween such entities.

FIG. 1 illustrates one exemplary mobile communications device 10 onwhich various embodiments of the present disclosure may be implemented.The mobile communications device 10 may be a smartphone, and thereforeinclude a radio frequency (RF) transceiver 12 that transmits andreceives signals via an antenna 13. Conventional devices are capable ofhandling multiple wireless communications modes simultaneously. Theseinclude several digital phone modalities such as UMTS (Universal MobileTelecommunications System), 4G LTE (Long Term Evolution), and the like.For example, the RF transceiver 12 includes a UMTS module 12 a. To theextent that coverage of such more advanced services may be limited, itmay be possible to drop down to a different but related modality such asEDGE (Enhanced Data rates for GSM Evolution) or GSM (Global System forMobile communications), with specific modules therefor also beingincorporated in the RF transceiver 12, for example, GSM module 12 b.Aside from multiple digital phone technologies, the RF transceiver 12may implement other wireless communications modalities such as WiFi forlocal area networking and accessing the Internet by way of local areanetworks, and Bluetooth for linking peripheral devices such as headsets.Accordingly, the RF transceiver may include a WiFi module 12 c and aBluetooth module 12 d. The enumeration of various wireless networkingmodules is not intended to be limiting, and others may be includedwithout departing from the scope of the present disclosure.

The mobile communications device 10 is understood to implement a widerange of functionality through different software applications, whichare colloquially known as “apps” in the mobile device context. Thesoftware applications are comprised of pre-programmed instructions thatare executed by a central processor 14 and that may be stored on amemory 16. The results of these executed instructions may be output forviewing by a user, and the sequence/parameters of those instructions maybe modified via inputs from the user. To this end, the central processor14 interfaces with an input/output subsystem 18 that manages the outputfunctionality of a display 20 and the input functionality of a touchscreen 22 and one or more buttons 24.

In a conventional smartphone device, the user primarily interacts with agraphical user interface that is generated on the display 20 andincludes various user interface elements that can be activated based onhaptic inputs received on the touch screen 22 at positions correspondingto the underlying displayed interface element. One of the buttons 24 mayserve a general purpose escape function, while another may serve topower up or power down the mobile communications device 10.Additionally, there may be other buttons and switches for controllingvolume, limiting haptic entry, and so forth. Those having ordinary skillin the art will recognize other possible input/output devices that couldbe integrated into the mobile communications device 10, and the purposessuch devices would serve. Other smartphone devices may include keyboards(not shown) and other mechanical input devices, and the presentlydisclosed interaction methods detailed more fully below are understoodto be applicable to such alternative input modalities.

The mobile communications device 10 includes several other peripheraldevices. One of the more basic is an audio subsystem 26 with an audioinput 28 and an audio output 30 that allows the user to conduct voicetelephone calls. The audio input 28 is connected to a microphone 32 thatconverts sound to electrical signals, and may include amplifier and ADC(analog to digital converter) circuitry that transforms the continuousanalog electrical signals to digital data. Furthermore, the audio output30 is connected to a loudspeaker 34 that converts electrical signals toair pressure waves that result in sound, and may likewise includeamplifier and DAC (digital to analog converter) circuitry thattransforms the digital sound data to a continuous analog electricalsignal that drives the loudspeaker 34. Furthermore, it is possible tocapture still images and video via a camera 36 that is managed by animaging module 38.

Due to its inherent mobility, users can access information and interactwith the mobile communications device 10 practically anywhere.Additional context in this regard is discernible from inputs pertainingto location, movement, and physical and geographical orientation, whichfurther enhance the user experience. Accordingly, the mobilecommunications device 10 includes a location module 40, which may be aGlobal Positioning System (GPS) receiver that is connected to a separateantenna 42 and generates coordinates data of the current location asextrapolated from signals received from the network of GPS satellites.Motions imparted upon the mobile communications device 10, as well asthe physical and geographical orientation of the same, may be capturedas data with a motion subsystem 44, in particular, with an accelerometer46, a gyroscope 48, and a magnetometer 50, respectively. Although insome embodiments the accelerometer 46, the gyroscope 48, and themagnetometer 50 directly communicate with the central processor 14, morerecent variations of the mobile communications device 10 utilize themotion subsystem 44 that is embodied as a separate co-processor to whichthe acceleration and orientation processing is offloaded for greaterefficiency and reduced electrical power consumption. In either case, theoutputs of the accelerometer 46, the gyroscope 48, and the magnetometer50 may be combined in various ways to produce “soft” sensor output, suchas a pedometer reading, or to function collectively as an inertialmeasurement unit (IMU). One exemplary embodiment of the mobilecommunications device 10 is the Apple iPhone with the M7 motionco-processor.

The components of the motion subsystem 44, including the accelerometer46, the gyroscope 48, and the magnetometer 50, may be integrated intothe mobile communications device 10 or may be incorporated into aseparate, external device. This external device may be wearable by theuser and communicatively linked to the mobile communications device 10over the aforementioned data link modalities. The same physicalinteractions contemplated with the mobile communications device 10 toinvoke various functions as discussed in further detail below may bepossible with such external wearable device.

There are other sensors 52 that can be utilized in the mobilecommunications device 10 for different purposes. For example, one of theother sensors 52 may be a proximity sensor to detect the presence orabsence of the user to invoke certain functions, while another may be alight sensor that adjusts the brightness of the display 20 according toambient light conditions. Those having ordinary skill in the art willrecognize that other sensors 52 beyond those considered herein are alsopossible.

With reference to the flowchart of FIG. 2, one embodiment of a methodfor producing a gesture-controlled augmented reality experience usingthe mobile communications device 10 will be described. None of the stepsof the method disclosed herein should be deemed to require sequentialexecution. The method begins with a step 200 of receiving a motionsensor input on a motion sensor input modality of the mobilecommunications device 10. The motion sensor input modality may includeat least one of the accelerometer 46, the magnetometer 50, and thegyroscope 48 and may further include the motion subsystem 44. As themobile communications device 10 moves, either by a deliberate motion ofa person holding it or by an inadvertent motion of a substantiallystationary person attempting to hold the mobile communications device 10steady, the camera 36 of the mobile communications device 10 movestogether with the mobile communications device 10 according to sometrajectory of motion. In a step 202, this trajectory of the camera 36may be calculated in response to the received motion sensor input. Forexample, a combination of the accelerometer 46, magnetometer 50, andgyroscope 48 may function as an inertial measurement unit (IMU) tomeasure the position, velocity, or acceleration of the camera 36, e.g. alens of the camera 36, relative to a relevant reference frame. Therelevant reference frame may be, for example, a world reference frame inwhich the earth is considered stationary or a vehicle reference frame inwhich a steadily moving airplane or other vehicle is consideredstationary.

Simultaneously, or otherwise correspondingly with the receipt of themotion sensor input, a visual input captured by the camera 36 isreceived in a step 204. The visual input may include image data (e.g.one or more still images or video feed) of a scene in which a user makesa gesture such as a motion or attitude/arrangement of the user's face,hand(s), leg(s), or body. The user may be a different person from theperson holding the mobile communications device 10 and may, for example,be several meters away from the mobile communications device 10. As theuser makes one or more gestures, the method proceeds with a step 206 oftranslating a gesture into a set of quantified values based on thereceived visual input and the calculated trajectory of the camera 36.Because the trajectory of the camera 36 is calculated, the user'sgesture can be accurately observed irrespective of the motion of thecamera 36. In this way, the camera 36 and inertial measurement unit(IMU) or motion sensor input modalities thereof may constitute a sensorfusion for accurately discerning the user's gesture. As a simpleexample, a vector representing the trajectory of the camera 36 may besubtracted from a vector representing a trajectory of a landmark in thereceived visual input in order to cancel the motion of the camera 36 andconstruct a plane of observation in the relevant reference frame. Thelandmark may be, for example, the user or a body party of the user suchas the user's hand(s). Angles of observation in the constructed planemay then be measured using initial and terminal positions of the camera36 and estimated distances to the landmark. Known vision algorithms,deep image processing, and/or machine learning may be used to recognizethe landmark and/or movement patterns thereof constituting a gesture. Insome cases, a person's body or body part may be preregistered as alandmark during a setup/calibration step in order to improve distanceestimation and recognition accuracy.

In addition to the motion sensor input and visual input used to quantifythe user's gesture, an auditory input captured by the microphone 32 ofthe mobile communications device 10 may also be received in a step 208.For example, the user or the person holding the mobile communicationsdevice 10 (if different) may issue voice commands, e.g. “START CAPTURE,”“STOP CAPTURE,” “START OVER AGAIN,” etc. Auditory input may includemusic. In a case where the auditory input is music, a signature such asa tempo (e.g. beats per minute) melody, and otherfeatures/characteristics of the music may be detected in a step 210 byany known methods. See, e.g. Gainza, M. & Coyle, E. “Time SignatureDetection by Using a Multi-Resolution Audio Similarity Matrix,” 122ndAudio Engineering Society Convention, May 5-8, May 5-8, 2007, Vienna,Austria, the entire disclosure of which is hereby wholly incorporated byreference.

With the user's gesture having been translated into a set of quantifiedvalues, the method continues with a step 212 of controlling an augmentedreality object within a three-dimensional virtual environment inresponse to a substantial match between the set of quantified values anda set of predefined values. The augmented reality object may be visuallyperceptible, e.g. a visual feature such as a drawn line or a virtualphysical object. In such cases, the augmented reality object may be seenin a visual display of the three-dimensional virtual environment (e.g.using a rendering engine that supports a high poly-count of 3D assetswith textures). Alternatively, or additionally, the augmented realityobject may be an auditory effect emanating from a specific locality invirtual space and perceivable on a loudspeaker (such as the loudspeaker34 of the mobile communications device 10), a haptic effect emanatingfrom a specific locality in virtual space and perceivable on a hapticoutput device (such as the touch screen 22 or a vibration module of themobile communications device 10), a localized command or instructionthat provides a link to a website or other remote resource to a mobilecommunications device 10 entering its proximity in virtual space, or anyother entity that can be defined in a three-dimensional virtualenvironment and perceivable by an application that can access thethree-dimensional virtual environment.

Controlling the augmented reality object within the three-dimensionalvirtual environment may include, for example, placing the augmentedreality object, removing the augmented reality object, moving theaugmented reality object, or otherwise transforming the augmentedreality object. As a simple example, a match between the quantifiedvalues translated from a user's gesture and a set of predefined valuesmay simply result in selection of a particular augmented reality objectand placement thereof. For example, in a case where the user's handgestures are to be used to create augmented reality objects, a databaseof augmented reality objects may be stored in association with sets ofpredefined values (e.g. gesture recognition descriptors) of handpositions. When a set of quantified values translated from a user'sactual hand position matches a set of predefined values in the database,the associated augmented reality object is selected and placed in thethree-dimensional virtual environment anchored to the user's hand. Thus,for example, the user may hold up his hand with palm outward facingforward and a virtual stop sign may be selected and placed in thethree-dimensional virtual environment at the position of the user'spalm, to be viewed on the display of the mobile communications device 10or other device that can view the virtual space. In this way,controlling the augmented reality object based on the user's gesture mayinclude selecting and positioning the object.

As a more complicated example, the set of predefined values may beindicative not only of a particular augmented reality object, but alsoof parameters associated with that object including its movement withinthe three-dimensional virtual environment. So, for example, a user maymake a palm-down, moving hand gesture indicative of dribbling abasketball. As in the case of the virtual stop sign, a set of quantifiedvalues translated from the user's hand position may match a set ofpredefined values in a database, this time associated with a virtualbasketball, resulting in selection and placement of the virtualbasketball in the three-dimensional virtual environment anchored to theuser's hand (e.g. underneath the user's hand). In this case, however,the set of quantified values is continually translated from a gesturedefining the user's upward and downward motion of his hand, and thesequantified values are matched to predefined values associated with aparameter of the virtual basketball such as bounce speed and/or bounceposition. In this way, controlling the augmented reality object based onthe user's gesture may further include, in addition to selecting andpositioning the object, moving the object and determining variousparameters of the object.

In some cases, the control of the augmented reality object in step 212may additionally be based on received audio input as described above inrelation to step 208, such as voice commands or music, and/or based on adetected signature of the music as described in relation to step 210.Thus, for example, an augmented reality object such as the virtual stopsign or virtual basketball may be removed upon receipt of an appropriatevoice command. As another example, a bounce speed parameter of thevirtual basketball may be set to match a detected tempo of music.

As explained above, the augmented reality object may be visuallyperceptible. The method may further include a step 214 of displaying theaugmented reality object on the mobile communications device 10 and/oran external device local or remote to the mobile communications device10. Displaying the augmented reality object may include displaying amovable-window view of the three-dimensional virtual environment on themobile communications device 10. That is, a portion of thethree-dimensional virtual environment may be displayed on the display 20of the mobile communications device 10 and the user of the mobilecommunications device 10 may adjust which portion of thethree-dimensional virtual environment is displayed by panning the mobilecommunications device 10 through space. Thus, the angular attitude ofthe mobile communications device 10, as measured, e.g. by the gyroscope48, inertial measurement unit (IMU), or sensor fusion of IMU with camera36, may be used to determine which portion of the three-dimensionalvirtual environment is being viewed, with the augmented reality objectbeing visible within the three-dimensional virtual environment when therelevant portion of the three-dimensional virtual environment isdisplayed. A movable-window view may also be displayed on an externaldevice worn on or near the user's eyes and communicatively linked withthe mobile communications device 10 (e.g. viewing glasses or visor). Asanother example, displaying the augmented reality object in step 214 mayinclude displaying a large-area view of the three-dimensional virtualenvironment on an external device such as a stationary display local tothe user. A large-area view may be, for example, a bird's eye view or anangled view from a distance (e.g. a corner of a room), which may providea useful perspective on the three-dimensional virtual environment insome contexts.

The method may further include a step 216 of outputting, on the mobilecommunications device 10, at least one of visual, auditory, and hapticfeedback in response to a substantial match between the set ofquantified values translated from the user's gesture and a set ofpredefined values. Such feedback may enhance the user's feeling ofinteraction with the three-dimensional virtual environment. For example,when the user successfully performs a gesture that matches with a set ofpredefined values and results in the control of an augmented realityobject, an audible chime, bell, spoken words, or other sound from theloudspeaker 34 of the mobile communications device 10 may indicate theuser's success. Thus, in some examples, the predefined values foroutputting feedback and the predefined values for controlling anaugmented reality object may be one and the same, and, in such cases, itmay be regarded that the substantial match results both in the controlof an augmented reality object and the outputting of feedback. In othercases, visual, auditory, or haptic feedback may include audioinstructions aimed at correcting an unsuccessful gesture produced by theuser. For example, the mobile communications device 10 may detect somemovement of the user's hand but the movement may be indistinct andresult in quantitative values that ambiguously match multiple sets ofpredefined values or match a designated set of predefined valuesassociated with an “error” result. In this situation, the mobilecommunications device 10 may audibly instruct the user accordingly, e.g.“Please perform the gesture more slowly.” In this case, the predefinedvalues used for determining a substantial match for purposes ofoutputting visual, auditory, or haptic feedback may be different fromthose predefined values used for determining a substantial match forpurposes of controlling an augmented reality object.

Lastly, it should be noted that various additional steps may occurbefore, during, or after the method of FIG. 2. For example, instructionsmay be provided prompting the user as to what gesture to perform inorder to generate an augmented reality object within a three-dimensionalvirtual environment. The instructions may, for example, be displayed astext and/or graphics on the display 20 of the mobile communicationsdevice 10 at startup of an application for producing agesture-controlled augmented reality experience or at any other time,e.g. during loading or at a time that the application is ready toreceive the visual input representing the user's gesture. With regard tosuch an application, it should be noted that additional preliminarysteps may include, for example, displaying a content initializationscreen, detecting software compatibility and/or hardware capabilityincluding onboard sensors (e.g. accelerometer 46, gyroscope 48,magnetometer 50, touch screen 22, microphone 32, camera 36), firing upsensors, optimizing sensor filters (e.g. sensor control switch, sensorfusion, sensor algorithm parameter optimization), initializing storyelements of a three-dimensional virtual environment (e.g. placinginitial augmented reality objects), and/or receiving an initial input onthe mobile communications device 10 to trigger the activation of agesture-controlled augmented reality experience. Activation of agesture-controlled augmented reality experience may include, forexample, initiating the collection and evaluation of the motion sensorinput, visual input, auditory input, etc. using a control switch. Otherexamples of additional steps include designating roles for a user whowill produce the gesture (e.g. “actor”) and a person who will hold andoperate the mobile communications device 10 (e.g. “controller”),providing role-specific instructions, calibration for gesturerecognition (e.g. registering a user's body or body part(s) aslandmarks), establishing third party connections to other mobile devicesas described in more detail below, and computing analytics and/orstoring relevant data from the user's experience, including the user'sgestures, controlled augmented reality objects, etc., for later use suchas sharing. Such processing and storing, as well as any processing andstoring needed for performing the various steps of the method of FIG. 2(e.g. storing predefined values), may be performed, e.g. by the centralprocessor 14 and memory 16. It is also contemplated that a portion ofthe processing and storing needed for performing the various steps ofthe method of FIG. 2 may be performed remotely from the mobilecommunications device 10, such as on a cloud server.

FIG. 3 shows an example of a gesture-controlled augmented realityexperience produced according to the method of FIG. 2. A graphical userinterface of an application running on the mobile communications device10 includes primarily a live view image similar to that of a camera'slive preview mode or digital viewfinder, i.e. the default still or videocapture mode for most smart phones, in which a captured image iscontinuously displayed on the display 20 such that the real world may beviewed effectively by looking “through” the mobile communications device10. In the example of FIG. 3, a walking person can be seen in thethrough image. In this example, the walking person is the user whosegestures will control the augmented reality experience produced inaccordance with the method of FIG. 2. This user may also be referred toas having the “actor” role. In the example of FIG. 3, the person holdingthe mobile communications device 10 is a different person and may bereferred to as having the “controller” role. The graphical userinterface on the display 20 may additionally include text instructionsfor producing the augmented reality experience (e.g. “POINT CAMERA ATACTOR”), which may overlay the through image on the graphical userinterface such that the through image may be seen “behind” theinstructions, or which may be displayed in a pop-up window or adedicated top, bottom, or side panel area of the graphical userinterface. In the case of an application for producing a gesturecontrolled augmented reality experience, such instructions may bedisplayed or not depending on design or user preference, e.g. every timethe application runs, the first time the application runs, or never,relying on user knowledge of the application or external instructions.Non-display modes of instruction, e.g. audio instructions, are alsocontemplated.

As the person holding the mobile communications device 10 points thecamera of the mobile communications device 10 at the user, the user'swalking gesture is translated into quantified values by the mobilecommunications device 10 in accordance with steps 200, 202, 204, and 206of FIG. 2. With the user's walking gesture thus quantified, asubstantial match between the quantified values (e.g. values indicativeof leg position in a world reference frame as a function of time) andpredefined values (e.g. values such as gesture recognition descriptorsstored in a database in association with augmented reality objects)results in placement of a virtual flower path 72 as an augmented realityobject relative to a landmark (e.g. the user's legs) in athree-dimensional virtual environment. The matching between thequantified values and predefined values may further determine the valuesof parameters associated with the virtual flower path 72 (e.g. color,flower type, grass/flower height, growth speed, growth delay, etc.). Asthe user changes walking direction and speed, the virtual flower path 72follows along in the three-dimensional virtual environment, growingwhere the user walked, perhaps after a brief delay set by theapplication and/or adjustable according to parameter values controlledby the user's walking gesture or other input (e.g. audio input). In thisway, an augmented reality object, i.e. the virtual flower path 72, iscontrolled within a three-dimensional virtual environment in response toa substantial match between the set of quantified values translated fromthe user's gesture and a set of predefined values. The three-dimensionalvirtual environment including the augmented reality virtual flower path72 may be displayed on the display 20 of the mobile communicationsdevice 10 according to a movable-window view as described above with theviewing angle, distance, etc. determined based on the motion sensorinput and visual input (e.g. sensor fusion of IMU with camera 36).

In the example of FIG. 3, a substantial match between the quantifiedvalues and predefined values results in placement of a virtual flowerpath 72 as an augmented reality object and optionally determination ofparameters associated with the virtual flower path 72. In addition,depending on the application, the substantial match between thequantified values and predefined values may also yield the selection ofthe virtual flower path 72 in the first place, for example, from amongother potential augmented reality objects or more narrowly from amongother similar path-style objects. Thus, a particular application mightbe limited to controlling only the virtual flower path 72, or limited tocontrolling path-style objects, or broadly capable of controlling manytypes of augmented reality objects. Such limitations may be fundamentallimitations of the application or may be user-configured selected in asettings menu or at startup.

FIGS. 4A and 4B relate to another example of a gesture-controlledaugmented reality experience produced according to the method of FIG. 2.As in the example of FIG. 3, a graphical user interface of anapplication running on the mobile communications device 10 includes alive view image in which a captured image is continuously displayed onthe display 20. In the example of FIGS. 4A and 4B, a person's hands canbe seen in the through image. In this example, the person whose handscan be seen in the through image is the user (“actor”) whose gestureswill control the augmented reality experience produced in accordancewith the method of FIG. 2. The graphical user interface on the display20 may additionally include text instructions for producing theaugmented reality experience as described above, and non-display modesof instruction are also contemplated.

As the person holding the mobile communications device 10 points thecamera of the mobile communications device 10 at the user, the user'shand gestures are translated into quantified values by the mobilecommunications device 10 in accordance with steps 200, 202, 204, and 206of FIG. 2. With the user's hand gestures thus quantified, a substantialmatch between the quantified values (e.g. values indicative of handposition in a world reference frame as a function of time) andpredefined values (e.g. values such as gesture recognition descriptorsstored in a database in association with augmented reality objects)results in placement of virtual butterflies 74 relative to a landmark(e.g. the user's hands) in a three-dimensional virtual environment. Asubstantial match may further inform selection of the virtualbutterflies 74 as the particular augmented reality object in the firstplace as described above in relation to the virtual flower path 72.

Similar to the case of the virtual flower path 72, the matching betweenthe quantified values and predefined values may further determine thevalues of parameters associated with the virtual butterflies 74 (e.g.color, type, size, flight speed, flight pattern, flying/landed state,etc.). As the user performs predesignated gestures with his hands (asdefined by predefined values associated with specific controls of theaugmented reality object), the virtual butterflies 74 flutter around thethree-dimensional virtual environment in response to the user'sgestures. For example, as shown in FIGS. 4A and 4B, the inwardly cuppedattitude of the user's hands may be translated into quantified valuesthat determine the placement of the virtual butterflies 74, e.g. withthe butterflies centered between the concave sides of the user's cuppedhands and only occasionally flying outside the area between the user'shands or to the back of the user's hands. Meanwhile, the distancebetween the user's hands may be translated into quantified values thatdetermine the size of the individual butterflies, which may be smalleras the user's hands are brought closer together as shown in FIG. 4A orlarger as the user's hands are brought farther apart as shown in FIG.4B. In this way, an augmented reality object, i.e. the virtualbutterflies 74, may be controlled within a three-dimensional virtualenvironment in response to a substantial match between the set ofquantified values translated from the user's gesture and a set ofpredefined values. The three-dimensional virtual environment includingthe augmented reality virtual butterflies 74 may be displayed on thedisplay 20 of the mobile communications device 10 according to amovable-window view as described above with the viewing angle, distance,etc. determined based on the motion sensor input and visual input (e.g.sensor fusion of IMU with camera 36).

It should be noted that the virtual butterflies 74 may flutter around ontheir own within specific bounds or according to specific patterns(including random patterns) even without any change in the user'sgesture or other input. That is, the augmented reality object itself maybe a three-dimensional animation or set of three-dimensional featuresthat move relative to each other according to the particular softwarecode associated with that object, possibly with reference to defaultvalues of certain parameters (e.g. flight speed) that can also be set bythe user's gestures. By the same token, the virtual flower path 72 ofFIG. 3 may, for example, move as if to sway in the wind, irrespective ofthe user's walking motion.

Values of parameters associated with the virtual butterflies 74 mayfurther be determined in accordance with other input besides the user'sgestures, such as auditory input as described above. As an example, thepattern of motion of the virtual butterflies 74 may be determined inaccordance with a music signature detected in step 210 of the method ofFIG. 2, or the flight speed of the virtual butterflies 74 may bedetermined in accordance with a similarly determined music tempo. Inthis way, the virtual butterflies 74 may “dance” between the user'shands in response to music.

FIGS. 5A-5F show additional aspects related to the example of FIGS. 4Aand 4B. Each of FIGS. 5A-5F shows the display 20 of a mobilecommunications device 10, on which a live view image of the user's handgestures is visible together with the augmented reality virtualbutterflies 74, e.g. according to a movable-window view of thethree-dimensional virtual environment. Also visible in FIGS. 5A-5F aresmall dots indicative of features that have been found according to amachine vision algorithm for quantifying the user's gestures. Thepositions of these dots relative to a relevant reference frame (e.g. aworld reference frame determined based on a sensor fusion of an IMU andthe camera 36 of the mobile communications device 10) are an example ofquantified values translated from the user's gestures.

In FIG. 5A, the user is moving one hand around the scene to trace apath. The virtual butterflies 74 are controlled based on the user'sgesture to follow the user's hand, e.g. anchored to the user's hand as alandmark by areal and/or landmark dot density. In FIGS. 5B and 5C, theuser is holding two hands out with cupped palms facing upward, with theuser's cupped palms facing more inward toward each other in FIG. 5C. Thevirtual butterflies 74 freely fly upward away from the user's palms inFIG. 5B while flying in a more confined area between the user's cuppedhands in FIG. 5C. In FIG. 5C, the virtual butterflies 74 are alsosmaller than in FIG. 5B as the user's hands are closer together asdescribed in relation to FIGS. 4A and 4B. In FIG. 5D, the user isholding a single hand out palm up. This gesture may, for example, summona single butterfly from among the virtual butterflies 74 to alight onthe user's hand. In FIG. 5E, the user is holding both hands out palmsup. This gesture may, for example, summon a group of butterflies fromamong the virtual butterflies 74 to alight on the user's hands. In FIG.5F, the user is making a grabbing motion. This gesture may, for example,allow the user to grab/enclose one or a cluster of the virtualbutterflies 74 within a closed hand or between closed fingers. Thegrabbed butterfly or butterfly cluster may then be moved around thescene as the user moves his closed hand and may thereafter be releasedat a different position. The user may clap his hands to disperse thevirtual butterflies 74, temporarily or in some cases signaling the endof the augmented reality experience and causing the virtual butterflies74 to be removed.

Table 1 shows examples of single-handed gestures that may be implementedfor controlling augmented reality objects, such as the virtualbutterflies 74 described above, in accordance with embodiments of thepresent disclosure.

Single-Handed Gesture Augmented Reality Object Control Cupped hand Holdobject (e.g. with virtual gravity implemented, the object may rest inthe cupped hand) Closing fist Grasp/Select object Hold open hand Summonobject palm upward Hold open hand Deselect object/Disengage hand (fordouble palm downward handed gestures) Moving closed fist Move selectedobject Open hand motion Deselect object and set to “action/interaction”upward mode (e.g. the object may continue its default motion) Open handmotion Deselect object and set to “disengaged” mode downward (e.g. theobject may stop moving or be removed) Dismissal (move Deselect object inits most recent mode prior open hand from side to selection to otherside once) Throw (fist to open) Deselect object and set to“action/interaction” mode (e.g. the object may continue its defaultmotion)

Table 2 shows examples of double-handed gestures that may be implementedfor controlling augmented reality objects, such as the virtualbutterflies 74 described above, in accordance with embodiments of thepresent disclosure.

Double-Handed Gesture Augmented Reality Object Control Two open hands,Select object for two-handed manipulation palms upward One open hand,other Deselect summoned object and summon a hand “dismisses” differentone Moving hands farther Dilate selected object apart Moving handscloser Contract selected object together Rotate hands clockwise Rotateselected object clockwise or or counterclockwise counterclockwise Clapopen hands Duplicate object (e.g. generate one together in each hand)Close fist and stretch Expand object either vertically or vertical orhorizontal horizontally One fist, one other Hold onto selected object(in fist), apply single handed gesture appropriate action to secondaryobject for [e.g. one fist, one “hold other hand [e.g. hold onto selectedobject open hand palm upward”] (in fist), summon new object in openhand] Raise both open hands, Deselect any selected object and set allpalms upward objects to “action/interaction” mode (e.g. the objects maycontinue their default motion) Lower both hands, Set all objects to“disengaged mode” (e.g. the palms upward objects may stop moving or beremoved) Open hand palm up, Set selected object to predefined mode show1-5 digits other hand enumerated 1-5 Two handed throw Set all selectedobjects to “action/interaction mode” (e.g. the objects may continuetheir default motion)

With reference to the flowchart of FIG. 6, one embodiment of a methodfor producing a gesture-controlled augmented reality experience using amobile communications device 10A in communication with an additionalmobile communications device 10B will be described. None of the steps ofthe method disclosed herein should be deemed to require sequentialexecution. In the context of the method of FIG. 6, the mobilecommunications device 10 described in relation to FIG. 2 is given thereference number 10A and may be referred to as the first mobilecommunications device 10A (though the term “first mobile communicationsdevice” by itself is not intended to imply the existence of a secondmobile communications device and may be used to describe a sole mobilecommunications device). Meanwhile, the previously described user whomakes the gestures (i.e. the “actor”) carries an additional mobilecommunications device 10B that may be substantially the same as themobile communications device 10, 10A and may be referred to as thesecond mobile communications device 10B. The second mobilecommunications device 10B may serve various functions, includingproviding the user (“actor”) with a visual display of thethree-dimensional virtual environment including augmented realityobjects, providing feedback, and collecting additional input from theuser that will be used by the first mobile communications device 10A tocontrol the augmented reality object.

The method begins with steps 600 through 610, which may be substantiallythe same as steps 200 through 210 of the method of FIG. 2. As notedabove, various additional steps may occur before, during, or after themethod of FIG. 2. Likewise, the same additional steps may occur before,during, or after the method of FIG. 6. Such steps may include, forexample, establishing third party connections to other mobile devices asnoted above. In the case of FIG. 6, establishing third party connectionsmay include establishing a connection to communicatively couple thefirst mobile communications device 10A with the second mobilecommunications device 10B. In this regard, the connection may be director indirect, for example, via cellular or other wireless networks andmay refer to mutual connection to a server. It should also be notedthat, similar to the method of FIG. 6, any processing and storing neededfor performing the various steps of the method of FIG. 6 (e.g. storingpredefined values), may be performed, e.g. by the central processor 14and memory 16. It is also contemplated that a portion of the processingand storing needed for performing the various steps of the method ofFIG. 6 may be performed remotely from the mobile communications device10, such as on a cloud server.

The method of FIG. 6 differs from the method of FIG. 2 in the receiptand transmission of information from and to the second mobilecommunications device 10B. In particular, the method includes a step 612of receiving a set of secondary quantified values from the second mobilecommunications device 10B, in addition to the quantified valuestranslated from the user's gesture in step 606. With the first mobilecommunications device 10A thus having received the set of secondaryquantified values from the second mobile communications device 10B, theaugmented reality object may be controlled by the first mobilecommunications device 10A at least in part based on the receivedsecondary quantified values. That is, unlike corresponding step 212 ofFIG. 2, step 614 of FIG. 6 may include controlling the augmented realityobject in response to a substantial match between the set of secondaryquantified values and the set of predefined values. In other respects,step 614 of FIG. 6. may be substantially the same as step 212 of FIG. 2.

The method of FIG. 6 continues with a step 616 of displaying theaugmented reality object on the mobile communications device and/or anexternal device local or remote to the mobile communications device 10,which may be substantially the same as step 214 of FIG. 2, and a step618 of outputting, on the mobile communications device 10, at least oneof visual, auditory, and haptic feedback in response to a substantialmatch between the set of quantified values translated from the user'sgesture and a set of predefined values, which may be substantially thesame as step 216 of FIG. 2. However, in step 618, the output of feedbackmay further be based on a substantial match between the set of secondaryquantified values received from the second mobile communications device10B and the set of predefined values.

The method of FIG. 6 further differs from the method of FIG. 2 in theaddition of a step 620 of transmitting object control data to the secondmobile communications device 10B, the object control data defining oneor more movements of the augmented reality object within thethree-dimensional virtual environment in response to the control of theaugmented reality object performed in step 614. The object control datamay further define changes to the augmented reality object (includingaddition or removal thereof) that are not movements, such asmodifications to various parameters of the augmented reality object asdescribed throughout the present disclosure. The object control data mayeffectively inform the second mobile communications device 10B of anyand all effects produced by the first mobile communications device 10Awith respect to augmented reality objects in the three-dimensionalvirtual environment.

Lastly, the method of FIG. 6 further differs from the method of FIG. 2in the addition of a step 622 of transmitting feedback data to thesecond mobile communications device 10B, the feedback data defining atleast one of visual, auditory, and haptic feedback to be output by thesecond mobile communications device 10B in response to a substantialmatch between the set of quantified values and a set of predefinedvalues. The feedback data may define the same or different feedback fromthat which is output by the first mobile communications device 10A instep 618. For example, the feedback output in step 618 may inform theperson (“controller”) holding the first mobile communications device 10Aof something relevant to holding the first mobile communications device10A (e.g. the camera 36 should be moved to better capture the “actor”),while the feedback to be output by the second mobile communicationsdevice 10B may provide instruction relevant to correctly performing agesture.

With reference to the flowchart of FIG. 7, a complementary method to themethod of FIG. 6, to be performed by the mobile communications device10B, will be described. None of the steps of the method disclosed hereinshould be deemed to require sequential execution. First, in a step 700,the second mobile communications device 10B receives the object controldata that was transmitted by the first mobile communications device 10Ain step 620 of FIG. 6. In steps 702 and 704, the second mobilecommunications device 10B further receives a motion sensor input on amotion sensor input modality of the mobile communications device 10B anda visual input captured by the camera 36 of the mobile communicationsdevice 10B. Steps 702 and 704 may be substantially the same as steps 600and 604 of FIG. 6, but with respect to the mobile communications device10B rather than the mobile communications device 10A. However, ratherthan using the motion sensor input and visual input to translate a usergesture into quantified values as in step 606 of FIG. 6, the method ofFIG. 7 takes advantage of the object control data received from thefirst mobile device 10A in step 700. Thus, it is not necessary torecognize the user's gesture, which has already been recognized by thefirst mobile device 10A and used to control the augmented reality objectand generate the corresponding object control data. Instead, the methodof FIG. 7 proceeds with displaying the augmented reality object on thesecond mobile device 10B (e.g. on a display 20 thereof) based on theobject control data received from the mobile communications device 10Aand one or both of the motion sensor input and visual input of themobile communications device 10B received in steps 702 and 704. Forexample, to display a movable-window view of the three-dimensionalvirtual environment as described above, the angular attitude of themobile communications device 10, as measured, e.g. by the gyroscope 48,inertial measurement unit (IMU), or sensor fusion of IMU with camera 36in the form of motion sensor input and visual input, may be used todetermine which portion of the three-dimensional virtual environment isbeing viewed, with the augmented reality object being visible within thethree-dimensional virtual environment when the relevant portion of thethree-dimensional virtual environment is displayed. As in the case ofthe first mobile communications device 10A, a movable-window view mayalso be displayed on an external device worn on or near the user's eyesand communicatively linked with the second mobile communications device10B (e.g. viewing glasses or visor).

It should be noted that the motion sensor input and visual inputreceived in steps 702 and 704 may, in principle, be used to recognizethe user's gesture. However, as the user performing the gesture isholding the second mobile communications device 10B, the visibility ofthe gesture from the perspective of the camera 36 of the second mobilecommunications device 10B may be impaired and accuracy of gesturerecognition may be improved through the use of spatially removed firstmobile communications device 10A. Moreover, by using the first mobilecommunications device 10A that is not held by the user performing thegesture, the camera motion unrelated to the gesture can more easily becanceled as described above.

In a step 708, the method of FIG. 7 proceeds with the second mobilecommunications device 10B receiving the feedback data that wastransmitted by the first mobile communications device 10A in step 622 ofFIG. 6. The first mobile communications device 10B may output visual,auditory, or haptic feedback in accordance with the received feedbackdata in a step 710.

In a step 712, the method of FIG. 7 may further include receivingauditory input captured by the microphone 32 of the second mobilecommunications device 10B.

Having collected various data on input modalities of the mobilecommunications device 10B in steps 702, 704, and 712, namely dataassociated with the user who is performing the gestures (the “actor”)and his/her immediate vicinity, the second mobile communications device10B derives secondary quantified values from the collected data in astep 714 and transmits the secondary quantified values to the firstmobile communications device 10A in a step 716 of the method. Forexample, the set of secondary quantified values may be derived from themotion sensor input on the motion sensor input modality of the secondmobile communications device 10B, the visual input captured by thecamera 36 of the second mobile communications device 10B, and/or theauditory input captured by the microphone 32 of the second mobilecommunications device 10B. The secondary quantified values transmittedby the second mobile communications device 10B in step 716 are receivedby the first mobile communications device 10A in step 612 of FIG. 6 andused by the first mobile communications device 10A to control theaugmented reality object as described in relation to step 614. In thisway, the first mobile communications device 10A may control theaugmented reality object based not only on the user's gesture and otherinputs as observed by the first mobile communications device 10A, butfurther based on various inputs (including those associated with thesame gesture) as observed by the second mobile communications device 10Bheld by the user.

FIGS. 8A-8C relate to an example of a gesture-controlled augmentedreality experience produced according to the method of FIG. 6 and thecomplementary method of FIG. 7. As in the example of FIG. 3, a graphicaluser interface of an application running on a mobile communicationsdevice includes a live view image in which a captured image iscontinuously displayed on the display 20. Note, however, that thismobile communications device is labeled 10A in the context of FIGS.8A-8C and may be referred to as a first mobile communications device 10Ain accordance with the methods of FIGS. 6 and 7. FIGS. 8A-8C furtherinclude a second mobile communications device 10B held by the user(“actor”) whose gestures will control the augmented reality experience.The graphical user interface on the display 20 may additionally includetext instructions for producing the augmented reality experience asdescribed above, and non-display modes of instruction are alsocontemplated.

As the person holding the first mobile communications device 10A pointsthe camera of the first mobile communications device 10A at the user,the user's walking gesture is translated into quantified values by themobile communications device 10A in accordance with steps 600, 602, 604,and 606 of FIG. 6. With the user's walking gesture thus quantified, asubstantial match between the quantified values (e.g. values indicativeof leg position in a world reference frame as a function of time) andpredefined values (e.g. values such as gesture recognition descriptorsstored in a database in association with augmented reality objects)results in placement of a virtual flower path 72 as an augmented realityobject relative to a landmark (e.g. the user's legs) in athree-dimensional virtual environment. As in the case of the aboveexample of FIG. 3, the matching between the quantified values andpredefined values may further determine the values of parametersassociated with the virtual flower path 72 (e.g. color, flower type,grass/flower height, growth speed, growth delay, etc.). As the userchanges walking direction and speed, the virtual flower path 72 followsalong in the three-dimensional virtual environment, growing where theuser walked, perhaps after a brief delay set by the application and/oradjustable according to parameter values controlled by the user'swalking gesture or other input (e.g. audio input). In this way, justlike in the example of FIG. 3, an augmented reality object, i.e. thevirtual flower path 72, is controlled within a three-dimensional virtualenvironment in response to a substantial match between the set ofquantified values translated from the user's gesture and a set ofpredefined values. Also, just like in the example of FIG. 3, thesubstantial match between the quantified values and predefined valuesmay yield the selection of the virtual flower path 72 in the firstplace. The three-dimensional virtual environment including the augmentedreality virtual flower path 72 may be displayed on the display 20 of themobile communications device 10A according to a movable-window view asdescribed above with the viewing angle, distance, etc. determined basedon the motion sensor input and visual input (e.g. sensor fusion of IMUwith camera 36).

The example of FIGS. 8A-8C differs from that of FIG. 3 in that the firstmobile communications device 10A is communicatively coupled with thesecond mobile communications device 10B held by the user to allow forcommunication between the two devices in accordance with the methods ofFIGS. 6 and 7. Thus, as the first mobile communications device 10Acontrols the virtual flower path 72 based on the user's gestures, thefirst mobile communications device 10A simultaneously transmitscorresponding object control data to the second mobile communicationsdevice 10B in accordance with step 620 of FIG. 6. Upon receiving thetransmitted object control data in step 700 of FIG. 7, the second mobilecommunications device 10B generates a display (step 706) of the samevirtual flower path 72 as controlled by the first mobile communicationsdevice 10A, but visible from the perspective of the second mobilecommunications device 10B (as the display may also be based on acombination of motion sensor input and visual input received in steps702 and 704). Thus, as shown in FIG. 8B, the user may turn around andview the virtual flower path 72 behind her using a movable-window viewof the three-dimensional virtual environment displayed on the secondmobile communications device 10B. In this way, as the user continues totake steps, the first mobile communications device 10A may control thecontinuing growth of the virtual flower path 72 based on the user'swalking leg gestures (step 614), display the virtual flower path 72 fromthe perspective of the first mobile communications device 10A for thebenefit of the controller (step 616), and transmit the object controldata to the second mobile communications device 10B (step 620) so thatthe second mobile communications device 10B can display the virtualflower path 72 from the perspective of the second mobile communicationsdevice 10B for the benefit of the actor (step 706).

In FIG. 8C, the user begins to shake the mobile communications device10B, causing the motion sensor input collected by the second mobilecommunications device 10B in step 702 to include data indicative of theshaking. Secondary quantified values derived from this motion sensorinput in step 714 and transmitted to the first mobile communicationsdevice 10A in step 716 therefore include values indicative of theshaking. Upon receiving such secondary quantified values from the secondmobile communications device 10B in step 612, the first mobilecommunications device 10A may then control the virtual flower path 72based on such secondary quantified values in step 614. For example, thesecondary quantified values may match a set of predefined values in adatabase associated with the values of one or more parameters of theaugmented reality object, i.e. the virtual flower path 72. In this way,by shaking the mobile communications device 10B, the user canparticipate in the control of the virtual flower path 72. In the exampleof FIG. 8C, the shaking of the mobile communications device 10B causesthe virtual flower path 72 to grow taller in the three-dimensionalvirtual environment. The user can immediately witness this effect as thefirst mobile communications device 10A transmits updated object controldata to be received by the second mobile communications device 10B andthe second mobile communications device 10B updates the display of theaugmented reality object.

In the example of FIGS. 8A-8C, the virtual flower path 72 is controlledprimarily by the user's walking gesture as recognized by the firstmobile communications device 10A, with some additional control describedbased on the user shaking the second mobile communications device 10B.However, it is also envisioned that the virtual flower path 72 may becontrolled wholly or partly based on recognition of the user's walkingmotion by the motion sensor input modalities of the second mobilecommunications device 10B held by the walking user. In this regard, theembodiments of the present disclosure may be supplemented or otherwisecombined with the techniques disclosed in co-owned U.S. PatentApplication Pub. No. 2017/0269712, entitled “IMMERSIVE VIRTUALEXPERIENCE USING A MOBILE COMMUNICATION DEVICE,” the entire disclosureof which is hereby wholly incorporated by reference.

In the examples described above, a single mobile communications device10 and a pair of mobile communications devices 10A, 10B are described.It is further contemplated that a network of mobile communicationsdevices 10 including three or more mobile communications devices 10 maybe communicatively coupled to control augmented reality objects inaccordance with the embodiments of the present disclosure. In this way,multiple sensors, cameras, microphones, etc. in a variety of locationscan be used to improve gesture recognition, implement additional sourcesof object control, and/or provide additional participation andcollaboration. It is further contemplated that the one or more mobilecommunications devices 10 may be communicatively coupled with othercomputing devices, e.g. a desktop computer or server. For example, adesktop computer may wait as a session with an augmented reality objectis recorded and thereafter provide playback or sharing features.Simultaneous streaming to a third-party device, e.g. a desktop computer,is also envisioned. In this way, processing and storage requirements ofthe mobile communications device(s) 10 may be reduced.

The above description is given by way of example, and not limitation.Given the above disclosure, one skilled in the art could devisevariations that are within the scope and spirit of the inventiondisclosed herein. Further, the various features of the embodimentsdisclosed herein can be used alone, or in varying combinations with eachother and are not intended to be limited to the specific combinationdescribed herein. Thus, the scope of the claims is not to be limited bythe illustrated embodiments.

What is claimed is:
 1. A method for producing a gesture-controlledaugmented reality experience using a first mobile communications device,the method comprising: receiving a motion sensor input on a motionsensor input modality of the first mobile communications device from auser; calculating a trajectory of a camera of the first mobilecommunications device in response to the received motion sensor input;receiving one or more images of a person other than the user captured bythe camera of the first mobile communications device; translating agesture of the person other than the user into a set of quantifiedvalues based on the received one or more images and the calculatedtrajectory of the camera, the set of quantified values being calculatedby a removal of the calculated trajectory from the received one or moreimages of the person other than the user; and controlling an augmentedreality object within a three-dimensional virtual environment inresponse to a substantial match between the set of quantified values anda set of predefined values.
 2. The method of claim 1, furthercomprising: displaying the augmented reality object on the first mobilecommunications device.
 3. The method of claim 2, wherein said displayingthe augmented reality object includes displaying a movable-window viewof the three-dimensional virtual environment on the first mobilecommunications device.
 4. The method of claim 1, further comprising:outputting, on the first mobile communications device, at least one ofvisual, auditory, and haptic feedback in response to a substantial matchbetween the set of quantified values and a set of predefined values. 5.The method of claim 1, further comprising: receiving an auditory inputcaptured by a microphone of the first mobile communications device;wherein said controlling includes controlling the augmented realityobject within the three-dimensional virtual environment in response tothe received auditory input.
 6. The method of claim 5, furthercomprising: detecting a signature of music in response to the receivedauditory input; wherein said controlling includes controlling theaugmented reality object within the three-dimensional virtualenvironment in response to the detected signature.
 7. The method ofclaim 1, further comprising: transmitting object control data to asecond mobile communications device held by the person other than theuser and communicatively coupled to the first mobile communicationsdevice, the object control data defining one or more movements of theaugmented reality object within the three-dimensional virtualenvironment in response to said controlling.
 8. The method of claim 7,further comprising: receiving a set of secondary quantified values fromthe second mobile communications device; wherein said controllingincludes controlling the augmented reality object within thethree-dimensional virtual environment in response to a substantial matchbetween the set of secondary quantified values and the set of predefinedvalues.
 9. The method of claim 8, wherein the set of secondaryquantified values is derived from a motion sensor input on a motionsensor input modality of the second mobile communications device. 10.The method of claim 8, wherein the set of secondary quantified values isderived from a one or more images captured by a camera of the secondmobile communications device.
 11. The method of claim 8, wherein the setof secondary quantified values is derived from an auditory inputcaptured by a microphone of the second mobile communications device. 12.The method of claim 1, further comprising: transmitting feedback data toa second mobile communications device held by the person other than theuser and communicatively coupled to the first mobile communicationsdevice, the feedback data defining at least one of visual, auditory, andhaptic feedback to be output by the second mobile communications devicein response to a substantial match between the set of quantified valuesand a set of predefined values.
 13. The method of claim 1, wherein themotion sensor input modality of the first mobile communications deviceincludes at least one of an accelerometer, a gyroscope, and amagnetometer integrated into the first mobile communications device. 14.A system comprising a non-transitory program storage medium readable bya first mobile communications device, the medium tangibly embodying oneor more programs of instructions executable by the device to performoperations for producing a gesture-controlled augmented realityexperience, the operations comprising: receiving a motion sensor inputon a motion sensor input modality of the first mobile communicationsdevice from a user; calculating a trajectory of a camera of the firstmobile communications device in response to the received motion sensorinput; receiving one or more images of a person other than the usercaptured by the camera of the first mobile communications device;translating a gesture of the person other than the user into a set ofquantified values based on the received one or more images and thecalculated trajectory of the camera, the set of quantified values beingcalculated by a removal of the calculated trajectory from the receivedone or more images of the person other than the user; and controlling anaugmented reality object within a three-dimensional virtual environmentin response to a substantial match between the set of quantified valuesand a set of predefined values.
 15. The system of claim 14, furthercomprising: the first mobile communications device; wherein the firstmobile communications device includes a processor or programmablecircuitry for executing the one or more programs of instructions. 16.The system of claim 15, further comprising: a second mobilecommunications device held by the person other than the user andcommunicatively coupled to the first mobile communications device;wherein the operations further comprise transmitting object control datato the second mobile communications device, the object control datadefining one or more movements of the augmented reality object withinthe three-dimensional virtual environment in response to saidcontrolling.
 17. The system of claim 16, wherein: the second mobilecommunications device transmits a set of secondary quantified values tothe first mobile communications device; and said controlling includescontrolling the augmented reality object within the three-dimensionalvirtual environment in response to a substantial match between the setof secondary quantified values and the set of predefined values.
 18. Thesystem of claim 17, wherein the second mobile communications devicederives the set of secondary quantified values from a motion sensorinput on a motion sensor input modality of the second mobilecommunications device, one or more images captured by a camera of thesecond mobile communications device, and/or an auditory input capturedby a microphone of the second mobile communications device.
 19. A mobilecommunications device operable to produce a gesture-controlled augmentedreality experience, the mobile communications device comprising: amotion sensor for receiving a motion sensor input from a user; a camerafor capturing a one or more images of a person other than the user; anda processor for calculating a trajectory of the camera in response tothe received motion sensor input, translating a gesture of person otherthan the user into a set of quantified values based on the received oneor more images and the calculated trajectory of the camera, the set ofquantified values being calculated by a removal of the calculatedtrajectory from the received one or more images of the person other thanthe user, and controlling an augmented reality object within athree-dimensional virtual environment in response to a substantial matchbetween the set of quantified values and a set of predefined values. 20.The mobile communications device of claim 19, further comprising: adisplay; wherein the processor controls the display to display theaugmented reality object within a movable-window view of thethree-dimensional virtual environment.